Adaptive Web Sites: Automatically Synthesizing Web Pages Adaptive Web Sites the Index Page Synthesis Problem the Pagegather Algorithm

Content Areas: data mining, machine learning, applications, user interfaces Abstract The creation of a complex web site is a thorny problem in user interface design. In IJCAI '97, we challenged the AI community to address this problem by creating adaptive web sites: sites that automatically improve their organization and presentation by mining visitor access data collected in Web server logs. In this paper we introduce our own approach to this broad challenge. Speciically, we investigate the problem of index page synthesis | the automatic creation of pages that facilitate a visitor's navigation of a Web site. First, we formalize this problem as a clustering problem and introduce a novel approach to clustering, which we call cluster mining: Instead of attempting to partition the entire data space into disjoint clusters, we search for a small number of cohesive (and possibly overlapping) clusters. Next, we present PageGather, a cluster mining algorithm that takes Web server logs as input and outputs the contents of candidate index pages. Finally , we show experimentally that PageGather is both faster (by a factor of three) and more eeective than traditional clustering algorithms on this task. Our experiment relies on access logs collected over a month from an actual web site. Designing a rich web site so that it readily yields its information can be tricky. The problem of good web design is compounded by several factors. First, diier-ent visitors have distinct goals. Second, the same visitor may seek diierent information at diierent times. Third, many sites outgrow their original design, accumulating links and pages in unlikely places. Fourth, a site may be designed for a particular kind of use, but be used in many diierent ways in practice; the designer's a priori expectations may be violated. Too often web site designs are fossils cast in HTML, while web navigation is dynamic, time-dependent, and idiosyncratic. In (Perkowitz & Etzioni 1997a), we challenged the AI community to address this problem by creating adaptive web sites: web sites that automatically improve their organization and presentation by learning from visitor access patterns. In this paper we report on our own progress. Sites may be adaptive in two basic ways. Customiza-tion is adapting the site's presentation to the needs of individual visitors, based on information about those individuals. In order to specialize itself to individual users, the site must maintain multiple copies of itself and gather quite a bit of information …